Skip to content

test: rewrite to e2e#2370

Open
bjohansebas wants to merge 47 commits into
hot-middlewarefrom
hot-e2e-tests
Open

test: rewrite to e2e#2370
bjohansebas wants to merge 47 commits into
hot-middlewarefrom
hot-e2e-tests

Conversation

@bjohansebas

Copy link
Copy Markdown
Member

Summary

What kind of change does this PR introduce?

Did you add tests for your changes?

Does this PR introduce a breaking change?

If relevant, what needs to be documented once your changes are merged or what have you already documented?

Use of AI

@changeset-bot

changeset-bot Bot commented Jul 25, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 3863108

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

bjohansebas and others added 8 commits July 25, 2026 17:53
… the console

The hot client, overlay, and building indicator are now exercised end
to end in headless Chrome (puppeteer 22, the last CJS release jest can
require) against a real webpack watcher: updates applied without
reload, the full-reload fallback, reconnect with catch-up sync after a
server restart, disconnect() semantics, overlay lifecycle (Escape on
the host page, HTML-in-error-message escaping, warnings, pagination),
the building badge, multi-compiler ?name= filtering with per-bundle
overlay slots, and the custom publish/subscribe API.

Console behavior is snapshotted from the real browser console: the full
info-level update cycle, logging=none/warn/error gates, and the
per-bundle dedup where a sibling's clean rebuild does not re-log
another bundle's unchanged warning while its own rebuild does. Each
multi-compiler compilation gets its own context dir so editing one
entry cannot invalidate the sibling's watcher and shuffle event order.

The jsdom suites stay, slimmed to what a browser cannot exercise
deterministically: process-update internals, shared state between
bundled copies, option parsing, and protocol edges. The review of the
new tests also surfaced a real client bug, now fixed: an EventSource
error event queued behind disconnect() re-armed the reconnect timer on
the orphaned wrapper — close() is now final.

e2e runs serially through the new test:e2e script (four concurrent
Chrome instances plus watchers starved the rest of the suite), and CI
shards every test job 4x with jest --shard, uploading per-shard
coverage that a final job aggregates for codecov, mirroring
webpack-dev-server's setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d unit tests

A logging=log e2e run now snapshots the collapsed "Updated modules:"
group and its per-module entries straight from the browser console
(puppeteer surfaces group frames as console events). The jsdom
equivalents — the process-update logging describe and the
building-message log test, whose output already appears verbatim in the
info-level e2e snapshot — are removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eir console

The inactivity watchdog is now exercised against a genuinely silent SSE
connection (a server heartbeat far beyond the client timeout): the
second connect proves it fires on pure silence and the third that it
re-arms after a reconnect. Manual recovery gets its own test —
disconnect() followed by setOptionsAndConnect() opens a fresh
connection that still delivers updates. Both, plus the server-restart
catch-up test, snapshot their console sequence; browser network-error
noise is filtered out of the snapshots since its volume depends on
reconnect timing.

The fake-timer equivalents in the jsdom suite are removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d copies

The multi-compiler helper already produces what the jsdom suite could
only simulate: two genuinely separate bundled copies of the indicator
module on one page. Each bundle now exposes its copy as a global and
the tests drive both from the browser — a second copy adopts the same
badge instead of stacking one, per-source counting spans copies, hiding
an unknown source is ignored, and a leaner state shape left by an older
package version gets its missing fields filled (planted via
evaluateOnNewDocument before the bundles load).

The jsdom indicator suite is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e jsdom suite

The subscribe/subscribeAll contract, the ?name= filter, per-source badge
counting, the shared SSE connection between bundled clients (asserted
explicitly in the multi-compiler console snapshot: one "connected" for
two clients), the dev-server-shaped overlay option, and the warnings
filter function — parsed from the entry query and applied against two
real warning-producing modules — now all run in Chrome. The warnings
overlay test also covers clearing on a recovered build, and the
pagination test snapshots the two-error console output that the deleted
jsdom snapshot used to document.

client.test.js keeps only what the browser cannot pin deterministically:
problem-type transitions, the multi-bundle problem union, option
forwarding to the overlay factory, dynamicPublicPath URL building, and
protocol edges.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Breaking both bundles shows the union of their problems in the shared
overlay — paginated one per page, in whichever order the compilers
finished. The two jsdom equivalents are removed: the union test, and
no-apply-on-errored-builds, which the multi-compiler console snapshot
already documents (no "Checking for updates" follows a bundle's error
report).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
webpack's invalid hook sometimes reports the watched directory instead
of the changed file under the polling watcher, so the "bundle
rebuilding (…)" line alternated between the file and the fixture dir
and flaked the snapshots. The normalizer now collapses both forms to
"(<fixture> changed)".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bundles' real hot clients call indicator.hide("a")/hide("b") when
their connect-time syncs arrive — the same source names these tests
register — so a late sync could wipe a source mid-test. progress=false
keeps the client away from the indicator entirely.
bjohansebas and others added 17 commits July 25, 2026 17:55
test/fixtures/js3 was compiled output (a misdirected copy of the
dev-server-false test's dist, whose real destination under test/outputs
is gitignored) swept up by a bulk git add.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--testPathIgnorePatterns is variadic, so a positional file after
`npm run test:only --` was consumed as another ignore pattern — silently
excluding the very file being targeted (the = form appends all the
same). The exclusion now lives in jest.config.js and lifts itself when a
test/e2e path is requested explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…copies

Same pattern as the indicator: each compilation exposes its own bundled
copy of the overlay module and the tests drive both from the browser —
a second copy adopts the shared iframe instead of stacking one, both
sources paginate together in the union, either copy can dismiss what
the other rendered, errors from one copy outrank another's warnings
until the erroring source recovers, clearing an unreported source does
not rebuild the card another copy is showing, and a leaner state shape
left by an older package version gets its missing fields filled.

The jsdom equivalents are removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backdrop click, close (×) button, and click-inside-stays-open now run
against real mouse events in the overlay iframe — the backdrop case
clicks the frame's top-left corner away from the centered card, and the
close-button pass rides the catch-up sync that re-shows the overlay
after a reload. The jsdom dismiss describe is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one-error-one-warning-one-* fixture configs build into
test/fixtures/js3 by design, so every logging run recreates it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pack 5.109

Runtime-error accumulation (two errors paging "1 / 2" ↔ "2 / 2") and
unhandled promise rejections now run against real page errors — the
rejection fixture rejects from the page's own script so the event
carries the real reason. Their jsdom equivalents are removed.

The branch also settles on webpack 5.109: the lockfile catches up with
the already-bumped range, and the assertions and console snapshots
adopt 5.109's parse-error format (numbered code-frame gutter, caret
line, and the "File was parsed as module type" note) instead of
normalizing it away.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…logger

webpack/lib/logging/runtime.js pulls tapable into the client bundle,
and tapable compiles its hooks with new Function — an EvalError under a
require-trusted-types-for 'script' CSP that killed the client's logging
and surfaced spurious runtime errors in the overlay. The replacement is
a level-gated console logger with byte-identical output (same prefix
merging, same level gates, groups from the "log" level up), verified
against the existing console snapshots. It also drops tapable and
webpack's logging machinery from the bundle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The page is served with require-trusted-types-for 'script' and a
trusted-types allowlist holding only the configured policy name — under
real Chrome enforcement, which jsdom cannot do, the overlay only
renders if every HTML write went through that policy (including inside
the about:blank iframe, which inherits the page's CSP). The jsdom
trusted-types test is removed; the helper gained a pageHeaders option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…time

Under a require-trusted-types-for 'script' CSP, tapable — bundled
through webpack's logging runtime — cannot compile its hooks: the first
log call throws an EvalError from new Function, killing whatever
listener happened to log (and with it the update flow). The logger
methods are now guarded, so webpack's logging runtime stays the engine
and enforcement just turns logging off.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…blish

Two browser pages connected to the same server both apply a single
edit — broadcast proven end to end. The unit variant also stops passing
vacuously: it now publishes while both SSE clients are attached instead
of only observing the per-client connect-time sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
waitForFunction's default requestAnimationFrame polling never fires on
a hidden document, so the backgrounded first tab of the broadcast test
hung on a wait for text it had already rendered. Also settle the
connections before editing so both pages receive the broadcast rather
than a catch-up sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… browser

invalidate() with unchanged sources rebuilds and reaches the page as a
building event followed by a no-op sync — the client neither re-renders
nor reloads (a built here would 404 on the never-emitted manifest).
close() ends the SSE stream under a connected page: the client falls
into its reconnect loop against an endpoint that no longer speaks SSE
and the page keeps running untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntime filter

Arrow keys page through problems with real keyboard events (the frame
holds focus after the pager click), paginate=false shows the full
problem list with no counter, and a runtime filter configured by a
later bundled copy is honored by the window listeners the first copy
attached — proven non-vacuously by first catching an error through
copy A, then watching copy B's rejecting filter keep the overlay away.
The three jsdom equivalents are removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The accent-bar colors are asserted (both hues, including the
error-to-warning flip) by the shared-state e2e, and the default
pagination counter by the pagination e2e.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…om describe

The pagination e2e now clamps at the last page under real clicks and
key presses, and a shared-state test drives a bundled copy through the
page-index semantics: a re-publish of the same problems keeps the page
the user navigated to, a different set starts back at page one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Problem replacement is asserted by the page-retention e2e (a new set
swaps the rendered content), and the overlay API shape is exercised by
every shared-state test driving real bundled copies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Escape e2e now checks the card advertises the dismissal it is about
to perform; the jsdom hint test is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bjohansebas and others added 7 commits July 25, 2026 21:06
The build-error e2e verifies webpack's own "See https://…" becomes a
safe new-tab link (href, target, rel), and the runtime-error e2e keeps
the sentence-ending dot out of the href. The two jsdom linkify tests
are removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The building payload's file and compiler name are pinned by the e2e
console snapshots, sync-vs-built by the invalidate and multi-compiler
runs, custom publishes by the subscribe round-trip, and the event
stream's delivery, broadcast, and close behavior by the two-page and
instance.close() tests. The invalidate e2e now snapshots its console —
a file-less building line and a sync silent enough not to log. What
stays in hot.test.js is the transport level: headers, heartbeat frames,
the headersSent guard, duplicate-name pairing, and progress throttling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hot.statsOptions warnings-filtering runs in the browser — its snapshot
is a lone "connected", the warning never entering the payload, while
the overlay={"warnings":false} snapshot shows the warning reaching the
console and stopping there. Three hot unit tests fall away: the
late-connect sync and post-close handling are covered by the e2e and
framework suites, and statsOptions forwarding by the new run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ates, and dynamicPublicPath

A warning overlay escalates to an error overlay when the build breaks,
an error overlay becomes a warning overlay on partial recovery (read
through the page realm so the reload fallback cannot detach it),
overlay={"runtimeErrors":false} leaves runtime errors uncaught, updates
carrying warnings apply without a reload, and the client connects and
updates through a dynamic public path served under /assets/ (the
helper gained a publicPath option). Six jsdom equivalents are removed —
client.test.js keeps only the slash-edge parsing and the protocol
guards nothing can reach end to end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A plain http reader against the running middleware covers what the
fake-response units simulated: the keep-alive handshake headers, real
heartbeat frames on real timers, and the catch-up sync reaching only
the newly connecting client while already-attached ones stay quiet.
The three fake-timer/fake-response equivalents leave hot.test.js; what
remains there has no end-to-end path (pure parsing, HTTP/2 headers,
forced pairing edges, progress throttling).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p race

The shared SSE connection writes its catch-up sync once — a sibling
bundle still evaluating when it arrives misses it, so a warning carried
by that sync reached the console on some runs and not others. The
widget now starts clean (an empty catch-up cannot vary the sequence)
and picks its warning up through a build the page observes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A multi-step browser test spent its whole 120s budget on a cold
Windows runner without any single wait hanging — first build, rebuild,
and reload each run several times slower there. dev-server runs all
its tests under a flat --test-timeout=400000; the e2e suites now use
the same cap, while the per-wait 30s timeouts keep catching real hangs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bjohansebas and others added 10 commits July 25, 2026 22:39
… suites

One helper now owns the text/overlay waits (all on interval polling —
only one of the five copies had received the hidden-page fix), the app
fixtures (accepted, unaccepted, warning-carrying, boom), the overlay
and indicator element ids, and the browser/app teardown that every
describe had cloned. Two snapshots move a warning column, following
the shared fixture's indentation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ElementHandle.click's scrollIntoViewIfNeeded pre-check stalled CDP on a
slow Windows runner (Runtime.callFunctionOn timing out). The overlay
iframe is fixed at the viewport origin, so a raw mouse click at (5,5)
lands on the backdrop without any evaluate round-trip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The overlay's file chip carries webpack's real file:line:column
reference, and clicking it lands a GET on an express route the test
mounts through the helper's new setup hook — the full documented
open-in-editor round trip, not just a fetch spy. Without an endpoint
configured no chip is rendered. The jsdom describe is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With a 100ms server heartbeat, several real 💓 frames cross the wire
and reach neither subscribers (only the catch-up sync does) nor the
console — snapshotted as a lone "connected". A rogue SSE endpoint
mounted through the helper's setup hook feeds the real EventSource a
non-JSON frame: the client warns (snapshotted, V8's wording is pinned
by the puppeteer-locked Chrome) and the page keeps running. The two
jsdom equivalents leave; client.test.js is down to the slash-edge
parsing and the no-EventSource guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ventSource

The dynamic public path is now asserted on the URLs the browser
actually requests: a bundled setter swaps __webpack_public_path__ at
runtime (the free variable only exists inside the bundle) and manual
reconnects show intentional double slashes surviving and trailing
slashes not doubling. Deleting EventSource before the bundle loads
(evaluateOnNewDocument) snapshots the polyfill warning with the page
still rendering. The jsdom client suite is gone — nothing was left
that a real browser could not cover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The silent watchdog keeps reconnecting by design, so a fourth
"connected" can land in the same poll window as the awaited third and
flake the full-list snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
publish() turns out to be the injection point the fake runtimes
simulated: a built announcing a hash the server never produced walks
the real check → manifest 404 → reload fallback; a sync carrying the
bundle's own hash (read through a bundled getter — __webpack_hash__
only exists inside the bundle) locks the compilation name so a
sibling's impossible hash is ignored, snapshotted as pure silence; and
without HotModuleReplacementPlugin the disabled-runtime error logs
exactly once across two builds. The accept-handler error path runs on
a genuinely throwing accept callback: reload by default, a warning and
the broken state kept with reload=false. Six fake-runtime units leave;
what stays needs forced runtime states no real flow reaches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The close-button click stalled on a Windows runner with the same
Runtime.callFunctionOn timeout as the backdrop before it — main-world
evaluates go unresponsive after navigations there. clickInFrame resolves
the element through the isolated-world selector wait, takes its
clickable point from the DOM domain, and clicks with raw mouse input;
the in-card and close-button clicks use it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sibling-sync race runs deterministically against the real runtime:
a slowed-down manifest route holds the sibling's check open while the
own sync locks the compilation name, and the failed check is discarded
in silence — snapshotted as a connect and a single "Checking". Trying
to force the runtime's failure status through a missing update chunk
surfaced the real behavior instead: check rejects with a
ChunkLoadError, no failure status is reached, and the client logs
"Update check failed" and stays put — now pinned as its own test. The
abort-status unit stays: no real flow reaches that state on demand.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The failure-status branch it exercised is unreachable through the real
runtime: the client's ignoreUnaccepted/ignoreDeclined/ignoreErrored
options close every path webpack 5.109 has to the abort and fail
statuses. The branch itself stays as a cheap guard for upstream
changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bjohansebas

Copy link
Copy Markdown
Member Author

I know this is a large PR. I can split it into smaller parts if that helps. Also, it currently only uses Express would you like us to test the other frameworks as well?

@bjohansebas
bjohansebas marked this pull request as ready for review July 26, 2026 05:11
…or real

A fixture function literally named invokeGuardedCallbackDev throws, so
the genuine stack carries the marker React's dev build leaves on
boundary-handled errors — the overlay ignores it while a plain error
right after proves the listeners live. The runtime slot resets through
its real trigger: a clean rebuild's reporter clears the accumulated
errors, and the next one starts a fresh slot with no pager. The jsdom
runtime-errors describe is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.70%. Comparing base (11f24ab) to head (3863108).

Additional details and impacted files
@@                Coverage Diff                 @@
##           hot-middleware    #2370      +/-   ##
==================================================
+ Coverage           94.04%   94.70%   +0.65%     
==================================================
  Files                   4        4              
  Lines                1226     1019     -207     
  Branches              373        0     -373     
==================================================
- Hits                 1153      965     -188     
+ Misses                 65       54      -11     
+ Partials                8        0       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bjohansebas and others added 4 commits July 26, 2026 00:21
The ERROR/WARNING badges (text and background) and the highlighted file
path are checked where Chrome actually renders them — on webpack's own
parse error and critical-dependency warning. Three jsdom rendering
tests leave.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The final overlay rendering details run in Chrome: the code-frame
highlight on webpack's real parse error, the re-mount after something
wiped the iframe (re-triggered by an unchanged rebuild, exercising the
identical-set guard), clear() as a no-op with nothing shown, and the
query-configured custom card styles and ansi colors driving the badge
and border. With no jsdom consumer left, jest-environment-jsdom leaves
the dev dependencies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migrating server behavior to the browser suites moved its execution out
of the coverage run — the heartbeat write, the SSE handshake, and every
hot.js path only e2e exercises showed as uncovered. The e2e step now
instruments src/ too (the middleware runs inside the jest process even
under a browser), writes to its own directory, and the artifact carries
both lcov files for codecov to merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
macOS fired a spurious startup rebuild whose unchanged hash broadcast a
legitimate sync to every client — indistinguishable from a re-sent
catch-up at the frame level. The helper's settle() drains builds until
the watcher goes quiet, and the catch-up test runs after it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant